Measuring and Improving Consistency in Pretrained Language Models
نویسندگان
چکیده
Abstract Consistency of a model—that is, the invariance its behavior under meaning-preserving alternations in input—is highly desirable property natural language processing. In this paper we study question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge? To end, create ParaRel
منابع مشابه
Automatic Implementation of Programming Language Consistency Models
Concurrent threads of execution running on a shared memory system can access the same memory locations. A consistency model defines constraints on the order of these shared memory accesses. For good run-time performance, these constraints must be as few as possible. Programmers who write explicitly parallel programs must take into account the consistency model when reasoning about the behavior ...
متن کاملImproving Context Aware Language Models
Increased adaptability of RNN language models leads to improved predictions that benefit many applications. However, current methods do not take full advantage of the RNN structure. We show that the most widely-used approach to adaptation (concatenating the context with the word embedding at the input to the recurrent layer) is outperformed by a model that has some low-cost improvements: adapta...
متن کاملImproving Japanese language models using POS information
In this paper, part-of-speech (POS) information is used to improve the performance of a Japanese language model (LM). The POS bigram is used to tackle the sparseness problem of the training data. Additionally, due to the characteristics of the Japanese language, part of the Japanese syntax information can be integrated into the POS bigram, through POS combination rules. Based on the Japanese sy...
متن کاملImproving Language Models by Clustering Training Sentences
Many of the kinds of language model used in speech understanding suffer from imperfect modeling of intra-sentential contextual influences. I argue that this problem can be addressed by clustering the sentences in a training corpus automatically into subcorpora on the criterion of entropy reduction, and calculating separate language model parameters for each cluster. This kind of clustering offe...
متن کاملImproving language models for radiology speech recognition
Speech recognition systems have become increasingly popular as a means to produce radiology reports, for reasons both of efficiency and of cost. However, the suboptimal recognition accuracy of these systems can affect the productivity of the radiologists creating the text reports. We analyzed a database of over two million de-identified radiology reports to determine the strongest determinants ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Transactions of the Association for Computational Linguistics
سال: 2021
ISSN: ['2307-387X']
DOI: https://doi.org/10.1162/tacl_a_00410